Embodiments according to the invention relating to the field of image processing and particularly an apparatus and a method for generating an overview image of the plurality of images.
Much research has been done in the area of mosaicking of aerial imagery and surveillance over the past years. Many approaches have been proposed ranging from using low altitude imagery of stationary cameras and UAVs (unmanned aerial vehicle) to higher altitudes imagery captured from balloons, airplanes, and satellites. High altitude imagery and on-ground mosaicking such as panoramic image construction are dealing with different challenges than low altitude imagery.
There has been a breakthrough regarding the seamless stitching in past years by exploiting robust feature extraction methods (see for example “Y. Zhan-long and G. Bao-long. Image registration using rotation normalized feature points. In ISDA '08: Proceedings of the 2008 Eighth International Conference on Intelligent Systems Design and Applications, pages 237-241, Washington, D.C., USA, 2008. IEEE Computer Society.”, “D. Steedly, C. Pal, and R. Szeliski. Efficiently registering video into panoramic mosaicks. In Proceedings of the Tenth IEEE International Conference on Computer Vision, volume 2, pages 1300-1307, Los Alamitos, Calif., USA, 17-21 2005. IEEE Computer Society.”, “H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-Up Robust Features (SURF). Comput. Vis. Image Underst., 110(3):346-359, 2008”), depth-maps (see for example “S. B. Kang and R. Szeliski. Extracting view-dependent depth maps from a collection of images. Int. J. Comput. Vision, 58(2):139-163, 2004”, “C. Cigla and A. A. Alatan. Multi-view dense depth map estimation. In IMMERSCOM '09: Proceedings of the 2nd International Conference on Immersive Telecommunications, pages 1-6, ICST, Brussels, Belgium, 2009. ICST Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering”), 3D reconstruction of the scene, image fusion, and many other approaches (e.g. “R. Szeliski. Image alignment and stitching: a tutorial. Found. Trends. Comput. Graph. Vis., 2(1):1-104, 2006”, “H.-Y. Shum and R. Szeliski. Construction and refinement of panoramic mosaicks with global and local alignment. In Proceedings of Sixth International Conference on Computer Vision, pages 953-956, 1998.”). A SURF feature-based algorithm is for example described in “H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-Up Robust Features (SURF). Comput. Vis. Image Underst., 110(3):346-359, 2008”. Results look seamless at the stitching part but a drawback is that the transformation performed on the images leads to a distortion in scales and relative distances. Lines which are parallel in reality, are not parallel anymore in the stitched image. This type of error accumulates over multiple images if not compensated. Such a traditional feature-based approach is difficult for some applications. For example, the generation of a geo-referenced image is hardly possible due to the scale and angle distortions as well as the error propagation over multiple images. Other stiching algorithms are shown in “H.-Y. Shum and R. Szeliski. Construction and refinement of panoramic mosaicks with global and local alignment. In Proceedings of Sixth International Conference on Computer Vision, pages 953-956, 1998”, “Y. Furukawa and J. Ponce. Accurate Camera Calibration from Multi-View Stereo and Bundle Adjustment. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, number 3, pages 1-8, Hingham, Mass., USA, 2008” or “G. Sibley, C. Mei, I. Reid, and P. Newman. Adaptive relative bundle adjustment. In Robotics Science and Systems (RSS), Seattle, USA, June 2009”.
A challenge of low altitude imagery and mosaicking for surveillance purposes is finding an appropriate balance between seamless stitching and geo-referencing under consideration of processing time and other resources. The scale difference as a result of different flying altitude resulted in several stitching errors. In other words, significant stitching errors induced by scale differences among images may be visible. Similar objects may have different sizes, and there may be a disparity in horizontal and vertical stitching. A similar error may occur by inaccurate camera position or rotation. In other words, stitching disparities may be caused by inaccurate camera angle or position.
Many approaches have been proposed to tackle these problems. Examples include the wavelet-based stitching “C. Yuanhang, H. Xiaowei, and X. Dingyu. A mosaick approach for remote sensing images based on wavelet transform. In WiCOM '08: Proceedings of the Fourth International Conference on Wireless Communications, Networking and Mobile Computing, pages 1-4, 2008”, image registering in binary domains “X. Han, H. Zhao, L. Yan, and S. Yang. An approach of fast mosaick for serial remote sensing images from UAV. In FSKD '07: Proceedings of the Fourth International Conference on Fuzzy Systems and Knowledge Discovery, pages 11-15, Washington, D.C., USA, 2007. IEEE Computer Society”, automatic mosaicking by 3D-reconstruction and epipolar geometry “L. Lou, F.-M. Zhang, C. Xu, F. Li, and M.-G. Xue. Automatic registration of aerial image series using geometric invariance. In Proceedings of IEEE International Conference on Automation and Logistics, pages 1198-1203, 2008”, exploiting known ground reference points for distortion correction “P. Pesti, J. Elson, J. Howell, D. Steedly, and M. Uyttendaele. Low-cost orthographic imagery. In GIS '08: Proceedings of the 16th ACM SIGSPATIAL international conference on Advances in geographic information systems, pages 1-8, New York, N.Y., USA, 2008. ACM”, IMU-based multi-spectral image correction “A. Jensen, M. Baumann, and Y. Chen. Low-cost multispectral aerial imaging using autonomous runway-free small flying wing vehicles. Geoscience and Remote Sensing Symposium, IGARSS, 5:506-509, 2008”, combining GPS, IMU and video sensors for distortion correction and geo-referencing “A. Brown, C. Gilbert, H. Holland, and Y. Lu. Near Real-Time Dissemination of Geo-Referenced Imagery by an Enterprise Server. In Proceedings of 2006 GeoTec Event, Ottawa, Ontario, Canada, June 2006” and perspective correction by projective transformation “W. H. WANG Yue, WU Yun-dong. Free image registration and mosaicking based on tin and improved szeliski algorithm. In Proceedings of ISPRS Congress, volume XXXVII, Beijing, 2008”. Some of these approaches are considering higher altitude “A. Brown, C. Gilbert, H. Holland, and Y. Lu. Near Real-Time Dissemination of Geo-Referenced Imagery by an Enterprise Server. In Proceedings of 2006 GeoTec Event, Ottawa, Ontario, Canada, June 2006”, “X. Han, H. Zhao, L. Yan, and S. Yang. An approach of fast mosaick for serial remote sensing images from UAV. In FSKD '07: Proceedings of the Fourth International Conference on Fuzzy Systems and Knowledge Discovery, pages 11-15, Washington, D.C., USA, 2007. IEEE Computer Society.”, “L. Lou, F.-M. Zhang, C. Xu, F. Li, and M.-G. Xue. Automatic registration of aerial image series using geometric invariance. In Proceedings of IEEE International Conference on Automation and Logistics, pages 1198-1203, 2008”, P. Pesti, J. Elson, J. Howell, D. Steedly, and M. Uyttendaele. Low-cost orthographic imagery. In GIS '08: Proceedings of the 16th ACM SIGSPATIAL international conference on Advances in geographic information systems, pages 1-8, New York, N.Y., USA, 2008. ACM”, “W. H. WANG Yue, WU Yun-dong. Free image registration and mosaicking based on tin and improved szeliski algorithm. In Proceedings of ISPRS Congress, volume XXXVII, Beijing, 2008”, while others are using different types of UAVs such as small fixed wing aircrafts “G. B. Ladd, A. Nagchaudhuri, M. Mitra, T. J. Earl, and G. L. Bland. Rectification, geo-referencing, and mosaicking of images acquired with remotely operated aerial platforms. In Proceedings of ASPRS 2006 Annual Conference, page 10 pp., Reno, Nev., USA, May 2006”, “A. Jensen, M. Baumann, and Y. Chen. Low-cost multispectral aerial imaging using autonomous runway-free small flying wing vehicles. Geoscience and Remote Sensing Symposium, IGARSS, 5:506-509, 2008.”, “Y. Huang, J. Li, and N. Fan. Image Mosaicking for UAV Application. In KAM '08: Proceedings of the 2008 International Symposium on Knowledge Acquisition and Modeling, pages 663-667, Washington, D.C., USA, 2008. IEEE Computer Society”. These aircrafts show less geo-referencing accuracy caused by higher speed and degree of tilting (higher amount of roll and pitch). “Z. Zhu, E. M. Riseman, A. R. Hanson, and H. J. Schultz. An efficient method for geo-referenced video mosaicking for environmental monitoring. Mach. Vis. Appl., 16(4):203-216, 2005” performed an aerial imagery mosaicking without any 3D reconstruction or complex global registration. That approach uses the video stream which was taken from an airplane. “Y. Huang, J. Li, and N. Fan. Image Mosaicking for UAV Application. In KAM '08: Proceedings of the 2008 International Symposium on Knowledge Acquisition and Modeling, pages 663-667, Washington, D.C., USA, 2008. IEEE Computer Society” performed a seamless feature-based mosaicking using a small fixed-wing UAV. “J. RoBmann and M. Rast. High-detail local aerial imaging using autonomous drones. In Proceedings of 12th AGILE International Conference on Geographic Information Science: Advances in GlScience, Hannover, Germany, June 2009”. also used small-scale quadrocopters. The mosaicking results are seamless but lacking geo-referencing.
“Howard Schultz, Allen R. Hanson, Edward M. Riseman, Frank Stolle, Zhigang Zhu, Christopher D. Hayward, Dana Slaymaker. A System for Real-time Generation of Geo-referenced Terrain Models. SPIE Enabling Technologies for Law Enforcement Boston, Mass., Nov. 5-8, 2000” and “Zhigang Zhu, Edward M. Riseman, Allen R. Hanson, Howard Schultz. An efficient method for geo-referenced video mosaicking for environmental monitoring. Machine Vision and Applications (2005) 16(4): 203-216” describes a system for generating 3D structures from aerial images useing laser sensor to precisely measure the elevations. For this, a larger airplane with two camera systems may be used.
In “M. Brown and D. G. Lowe. Recognising Panoramas. In Proc. ICCV 2003” a purely image-based mosaicking is shown. It describes an automatic approach for feature (using SIFT) and image matching by assuming that the camera rotates about its optical center.
“WU Yundong, ZHANG Qiangb, LIU Shaoqind. A CONTRAST AMONG EXPERIMENTS IN THREE LOW-ALTITUDE UNMANNED AERIAL VEHICLES PHOTOGRAPHY: SECURITY, QUALITY & EFFICIENCY. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Vol. XXXVII. Part B1. Beijing 2008” describes experiments for covering larger areas with UAVs and generating an overview image.
There are some software tools for image stitching available. However, they have several restrictions/assumptions (camera position and orientation, distance to objects etc.). For example, AutoPano (http://www.autopano.net/) takes a set of images and generates an overview (mosaick) which is visually most appealing. AutoPano stitches for beauty, at all areas where are less images or less overlap the distortions are still high with AutoPano. Another tool is PTGui Pro (http://www.ptgui.com/). PTGui stitches most panoramas fully automatically, but at the same time provides full manual control over every single parameter.